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# During the past 2 1/2 years-^he Office of Consumer Health Education, 
College of Medicine and Dentistry, o»f New Jersey, has been involved in 
evaluation of numerous health education programs. Brief descriptions of 
seven evaluative efforts follow*, along with some -considerations of problems 
-encountered. This is followed by two ''more detailed case studies. 

Brief Descriptions of Evaluative Efforts and Outcomes 

1. Lata from the first year of a veneral disease (VD) hotline at Momnouth 
Medical Center showed that, although there was a substantial increase in* 
visits to the clinic and emergency room and a modest increase in 4:reated 
cases, the causal contribution of the hotline to the increase could not be 
stated with certainty. The cost per call to operate the hotline was ex- 
cessive and could be reduced by making the hotline -serve for multipjLej 
health problems. A report of this evaluation appears in the May-June 1976- 
issue of Public Health Reports. 3 

2. A brief study of diabetes classes also at Mo'nmouth Medical Center revealed 
some weaknesses in the data collection process" and in certain areas of 
instruction. The program has been modif J^ed in an attempt to correct these 
problems. Further evaluation will be attempted at a later date. 

3. The Department of Community Health Education at the Medical Center meas- 
ured the outcomes of a Smoke-No-More program after 6 months, for two 
groups usingrdif f erent approaches — one a "soft sell" and the other a "hard 
sell." One-third ceased smoking in the former group and 17 percent, in 
the latter. 



ERLC 



Presented at the National Conference on ^Hospital-Based Patient Education, 
August 9-10, 1976, Chicago, Illinois, 'sponsored jointly by the American 
Hospital Association (840 N. Lake Shore Dr., Chicago, Illinois 60611) and 
the U.S. Department of Health, Education, and Welfare, Public Health 
Service, Center for Disease Control, Bureau of Health Education (Atlanta, 
Georgia 30333) . 5 • ^ , ^ 

2 

Director, Office of Consumer Health Education, College of Medicine and 
Dentistry of New Jersey, Piscataway, New Jersey 08854. 

^Bryant, N.H., Stender, W.^ Frist, W,, Somers, A.R. VD hotline: an evalu- 
ation. Public Health Reports 91(3): 231-235, May-rJune 1976.^ 



4. The outcomes of patient teaching in the diabetic clinic at Morris town 
Memorial Hospital were measured using weight control and blood sugar as 
independent variables, \lhen compared with a control group that did not 
have similar instruction, no statistical difference was revealed^ though 
there is some question as to whether the particular variables selected 
for measurement were appropriate for determining the success of a health 
education program. 

5. St. Francis Medical Center, Trenton, evaluated its Health Carnival td^ 
determine its value in promoting health actions. Six months follow-up 
of health screenings revealed that: 

a. 30 percent who had elevated blood sugars saw physicians; 

b. 60 percent who failed the hearing test were followed in the 
hospital *s clinic; 

c. 72 percent with elevated blooj pressures saw their physicians; 

d. 30 percent of the total number of senior .citizens who sighed up 
for flu shots went on the appointment day to receive the shots. 

6. An experimental summer health education program for 35 12-year-olds during 
1974 was followed up by a 1-day evaluation session 4 months later, at- 
tended by the students, their parents, and the project instructor. There 
was generally a high correlation among the parent and student respondents 
to the evaluation concerning the effectiveness of the program, A change 
in ^eating patterns of snacks - to fruit and huts from the usual chips and 
soda reflects-.a major success in a portion of the program that 'was 
highly stressed. I't'also shows self -direction in. health matters by the 
children since snacks are almost entirely under their control while other 
meals may not be. / - 

The most ambitious evaluation project undertaken by Office of Consumer 
Health Education involves patient teaching at Shore Memorial Hospital. A 
retrospective study has been conducted with the cardiac patients of a 
group practice at Mainland Medical Center who have received teaching from 
a nurse-health educator, including a planned educational program while 
hospitalized as well as reinforcement at subsequent office visits. These 
patients were compared with a control group of patients of other physi- 
cians in the same^area. The evaluation is designed to determine health 
education effects on risk factor reduction, modification of lifestyle, 
and understanding of the disease process as it applies to therapy. Pre- 
liminary findings from the retrospective study indicate, a reduced hospital 
readmission rate in the experimental group as compared with the control 
group . 
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Some Problems Encountered in Evaluation 

Our attempts to evaluate programs have brought into focus a specific set 
of problems and obstacles : 
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!• While objectives of programs are usually stated cleanly, they are 
often not measurable, especially in terms of costs or behavior 
mpdif ications . 

2. Obtaining the necessary data for evaluation purposes is essential.^ 
Unless the evaluation procedure is carefully planned ahead of time, 
the appropriate data may not be available or, if available, may be 
incomplete or even inaccurate. 

3. Another concern has to do with data collection — who will collect it, 
and how will it be paid for. Unless a specific person is responsible 
for evaluation data with adequate financial means to carry through 
the project, the importance of evaluation will probably not be rec- 
ognized. Thorough knowledge of the purposes and plan of evaluation 
will help insure cooperation from involved staff members. 

A. Long term- follow up represents an extremely difficult and sometimes 
impossible task in some stucjies. There may be no way to follow up 
patients or it may be i^ipractical to do so. While there may 
be an eagerness to show some results as soon as possible, many 
programs may not show results for years. 

5. The factor of cost is extremely important in all evaluations. A 
specific amount should be included in .the program budget if more than 
jus't the answering of a questionnaire is planned. Staff personnel 
often do not have the time nor the expertise to plan and carry out a 
cdmplete evaluation. Outside consultant help may have to be pai'd 
for as well as part-time students to collect and tabulate data. In a 
more elaborate study, costs for computer time should be included. 
Cost is one of the biggest hinderances to long term follow up in 
evaluation, which must be realistically considered in planning such 
studies. 

6. ^Population size is another consideration. Many programs do not 

have large enough numbers of participants to make them statistically 
significant. Higher numbers may be reached in some programs after . 
they have been repeated several times over a period of a year or 
two. Even then, with changes in program structure and evaluation 
instruments, results are delayed. 

7. Along with this is the problem* of findiag a control population, 
which is often expensive, time consuming, and in some cases imprac- 
tical or impossible. 

# 

8. One final concern has 'to do with the results and what is done with 
them. A major purpose of evaluation is educational-^-it shows weak- 
ness in the process of a program and indicates corrections needed. 
Another purpose is to examine program outcomes and measure effective- 
ness. The evaluation is not meant to point a finger but to help in 
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further development and changes in the programs. No one wants poor 
results, however, and some institutions are reluctant to release re- 
sults that show weaknesses. Care should be taken, therefore, in 
^ explaining the reasons and need for evaluation while stressing the 

non-threatening and constructive aspects. 

Case Study T. — Process Evaluation of Preopeifative Teachin g'^ 

^ A preoperative patient teaching program was begun in January 1974 
at Our Lad: o£ Lourdes Hospital, a 400-bed community hospital in Camden, 
New Jersey. The goals of the program were to reduce psychological stress, 
lessen postoperative discomfort, and hasten recovery by providing adequate 
preoperative instruction as a right of each surgical patient. 

Original program organization centered on a classroom situation held 
rwice each evening. This "method proved unsatisfactory when the results of 
a S-^month trial period revealed that only 50 percent of the 20 to 30 patients 
scheduled daily for surgery attended classes. The teaching program was 
changed to the one-to-one teaching method now being used, where all patients 
scheduled for surgery v/ere approached by nurse instructor^- Details of pro-^ 
cedures that would take place before, during, and after surgery were discussed. 
Simple breathing and leg exercises were demonstrated, and patients and family 
members- were^encoutaged to ask questions . 

^^^-^ At the end of the program's first year an evaluation questionnaire was 
distributed for patients to complete. Eaoh'^questlonnalre was distributed and 
picked up by the patient education coordinator several days postoperatively 
but before the patient was discharged. It included questions concerning the 
emotional support received by the patients from the instructors while they 
gave factual information and answered questions. Other questions were de- 
signed t-o determine the program's value and effectiveness. Excluded frgm 
the study were pediatric, cardiac, and local anesthesia patients. Using a 
table of random numbers, six patients were surveyed daily until a total of 
100 questionnaires were completed. The purpose of the evaluative <iuestion- 
naire was to determine if the preoperative teaching program was being carried 
out properly and if it was meeting any of its original objectives. 

The findings indicated that the , patients and their families found 
the program helpful and wanted it continued. It is important that the 
hospital administrators be aware of this fact, since the program represents 
^n additional cost in the hospital budget — although minimal when considering 
the average per diem cost. What cannot be measured is the amount of good 
will toward the hospital engendered by the program. 

Results of the questionnaires indicated the need to intensify the in- 
struction in the area of postoperative exercises, and steps have been taken 
to accomplish this goal^ 

''"Hannaberv, J. Process evaluation of preoperative teaching. First International 
Congress in Patient Counselling, Amsterdam, April 1976. 
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The evaluation sought to determine if at l^ast one of the program's 
objectives was met: namely, does preoperative teaching allay fear a and 
anxiety. Answers to related questions in^dicated that a large majority of 
patients were more relaxed after teaching and that over one-third had reduced 
anxiety. It is also understood that for some patients the effects^ of stress 
reduction was related, perhaps, more to the attitude and presence of the 
instructor than to the information and teaching. 

Although 77 percent of the respondents indicated that they read and 
understood the sjargical consent f orm,_ it is felt that__this_ is highly un- 
likely^ be cause of the detailed and complex nature of the forms. 

The inherent limitations of a subjective evaluation such as the one 
described here^ar^, recognized. It is generally acknowledged that respondents 
tend to offer the answers they think the investigators want to hear. This 
tendency may be even more pronounced among respondents who, like the surgery 
patients at Our Lady of Lourdes, have recently undergone a rather trying 
ordeal. 'Patients may 'fear appearing "ungrateful" to those who they believe 
have sincerely tried to help them through a difficult situation. 

In an effort to overcome this bias, the patient education coordinatgr 
rather :han the nurse-instructors—conducted th^ survey. The coordinator 
had neither participated in the preoperative teaching nor had any other 
prior contact with the patients surveyed. It was hoped that, if the coordi- 
nator distributed and explained the questionnaires, the patients* responses 
would reflect their ^ue feelings rather than a "desire to please." Un- 
fortunately, the degree to which this strategy succeeded is unknown. 

The evaluation of the preoperative teaching program at Our Lady of 
Lourdes did not utilize a control gr'^upl The hospital recognized that this 
would limit the usefulness of the survey's fi^idings. Nevertheless, it was 
decided against using a control group because this would deny^what was seen 
as an essential health service — preoperative teaching — to those patients 
selected as controls. 

Under consideration now' is the possibility of undertaking this type of 
survey again, perhaps, in cooperation with another similar hospital that 
does not offer preoperative teaching. The surgical patients in this hospical 
carefully matched to those at Our Lady of Lourdes would then serve as the 
controls. 

In conclusion, although no statistical analysis is possible from this 
kind of evaluation, the hospital and the personnel involved in the teaching 
feel that the questionnaire results indicate that the program is reaching its 
\ goals . 



Case Study II — Retrospective Study of Visiting Nurse Association Use by 
Patients with Diabetes and Ostomies 

The Coordinator of Consumer Health Education at Perth Amboy General 
Hospital (PAGE) heard a remark made by the;Director of t^e County Visiting 
Nurse Association (VNA) that her nurses seemed to make fewer, visits to PAGH 
patients who had received education before discharge. 

As a result of this remark, the evaluation person at the Office of 
Consumer Health Education (OCHE) is presently working with fhe health educa- 
tion coordinator at (PAGH) on a trend analysis utilizing the records of the 
VNA in Middlesex County. The overall objective of the evaluation is to de- 
termine if discharged cancer patients receiving a colostocry or discharged 
diabetic patients requiring insulin injections received fewer visits from 
the VNA in iliddlesex County than patients treated for the sam^ conditions 
but discharged from four other area hospitals not having a coordinated patient 
education program. It is anticipated that patients receiving patient education 
while hospitalized at (PAGH) required fewer visits by the VNA than patients 
who were treated for the same conditions at the other four "hospitals. By com- 
paring the number of visits to ostomy and insulin dependent diabetic patients 
from all five county hospitals, it is expected that we can develop a relative 
effectiveness index of the PAGH program. From these data, we will calculate a 
cost effectj^teOindex which could produce strong evidence oi cost effectiveness 
of the program, at PAGH as compared with the other four hospitals — some of 
which have discharge planning programs. OCHE is presently in whe process of 
collecting the data. 



